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Abstract 

This article discusses the evaluation of programmatic interventions to enhance postsecondary STEM education for 
students with disabilities. SciTrain University, a federally funded project to provide instructor training on acces¬ 
sible teaching according to universal design principles, is presented here as a case study on evaluation for similar 
programs. This article highlights the evaluation process, including relevant evaluation questions and selection of 
indicators, use of a mixed-methods approach, and development of instruments. We give particular attention to the 
utilization of longitudinal participants in order to document the project’s effectiveness. Project evaluators used 
demographic and performance data, classroom observations, and online journals to determine the efficacy of the 
training provided to these longitudinal participants. For these three activities, we discuss the development and de¬ 
ployment of instruments and offer some preliminary findings. Finally, the article concludes with a brief discussion 
of internal project assessment and its role in project refinement. Given a dearth of scholarship on the evaluation 
of programs to enhance STEM education for students with disabilities, this article seeks to provide some practical 
insights into the potential for mixed-methods approaches. 
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In recent decades, policymakers and educational 
leaders have emphasized accountability and utilization 
of evidence-based practices in secondary and postsec¬ 
ondary education (Fayzell, 1999; Elarvey & Williams, 
2010; Shin, 2010). Elowever, researchers have pointed 
to a continued lack of data on postsecondary students 
with disabilities and the need for appropriate indicators 
to document their success (Burke, Hedrick, Ouelette, & 
Thompson, 2008). Of particular concern is the efficacy 
of programs to improve enrollment in, retention within, 
and completion of science, technology, engineering, 
and mathematics (STEM) degrees at the postsecond¬ 
ary level. Projects sponsored by the National Science 
Foundation’s (NSF) Research in Disabilities Education 
(RDE) and the U.S. Department of Education’s Office 
of Postsecondary Education (OPE) have prioritized the 
improvement of postsecondary STEM education for 
students with disabilities (Burgstahler & Bellman, 2009; 


Izzo, Murray, & Novak, 2008; Stefanich, Gabrielle, 
Rogers, & Erpelding, 2005; Stumbo, Lindahl-Lewis, 
& Blegen, 2008). 

The need for more evidence-based practices has 
been noted in practically every aspect of disabilities 
education, including the transition to postsecondary 
education (Webb, Patterson, Syverud, Seabrooks- 
Blackmore, 2008). As the National Council on Dis¬ 
ability (NCD) has observed, “the amount of rigorous, 
evidence-based research on programs that promote 
positive outcomes for students with disabilities is 
severely limited” (Frieden, 2004, p. 6). In addition, 
there is a dearth of scholarship on how such programs 
might be evaluated. 

In response, this article presents SciTrain Uni¬ 
versity (SciTrain U), an OPE-fimded demonstration 
project at the Georgia Institute of Technology and 
University of Georgia designed to enhance the abilities 
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of STEM faculty to make instruction more accessible 
for students with disabilities through a combination of 
in-person and web-based training (Utschig, Moon, Todd, 
& Bozzorg, 2011). Using SciTrain U as a case study, we 
discuss methodologies for evaluating programs to sup¬ 
port STEM education for students with disabilities. In 
the course of highlighting key evaluation findings, this 
article presents indicators for determining programmatic 
efficacy, considers methodologies for gauging project 
performance, and discusses challenges faced by the 
project and efforts to resolve those difficulties. 

Background Discussion 

Scientific leaders and policymakers have called 
for the cultivation of a diverse STEM workforce in 
the United States (National Science Foundation, 1996, 
2000, 2004). This concern was reiterated by the Na¬ 
tional Science Board in its 2010 report, “Preparing the 
Next Generation of STEM Innovators,” which offers 
two interrelated observations. First, American prosper¬ 
ity in STEM in the coming years will rely increasingly 
upon “talented and motivated individuals who will 
comprise the vanguard of scientific and technologi¬ 
cal innovation” (National Science Board, 2010, p. 1). 
Second, every student in the United States “deserves 
the opportunity to achieve at his or her full potential” 
(National Science Board, 2010, p. 1). In summary, 
excellence and equity in STEM education go hand- 
in-hand. However, this goal can be realized only if 
underrepresented groups attain a larger proportion of 
the nation’s STEM degrees. 

Americans with disabilities historically have been 
underrepresented in postsecondary STEM education, 
particularly because these students face tremendous 
barriers to access and participation in these programs 
(Burgstahler, 1994; Wolanin & Steele, 2004). Participa¬ 
tion of students with disabilities tends to decrease longi¬ 
tudinally throughout the STEM education process. U.S. 
census data have shown that people with disabilities 
constitute 10% of the nation’s general workforce, but 
only two percent of its STEM professionals (Commit¬ 
tee on Equal Opportunities in Science and Engineering, 
2006; National Center for Education Statistics, 2004; 
U.S. Department of Education, 2001). 

The problem is complex. First, teachers, instruc¬ 
tors, and professors are frequently unable, unprepared, 
or otherwise ill-equipped to recognize and address the 
needs of students with disabilities (Stefanich, 2007). As 


a result, course content may be inaccessible, as many 
faculty fail to develop their courses in accordance with 
the principles of universal design for learning (UDL) 
(Burgstahler & Cory, 2008; Rose & Meyer, 2006; 
Rose, Meyer, & Hitchcock, 2005). Instructors may 
not be aware of strategies or technologies to help them 
accommodate students, or they may lack the necessary 
institutional support or resources to make accessible 
pedagogy a reality (Stefanich, 2001). In addition to 
the issue of instructional practice, there is a second 
matter of social inclusion. Research has demonstrated 
that students with disabilities, particularly learning 
disabilities, frequently experience negative attitudes 
from faculty and peers (Stage & Milne, 1996). By the 
time some of these students reach college, they are 
commonly discouraged from pursuing STEM degrees. 
When they enroll in STEM courses, many are not fully 
included in more rigorous learning activities such as 
labs, thus diminishing their potential engagement and 
prospects for success (DO-IT Staff, 2001). As such, 
there remains a pressing need for resources to ensure 
that STEM instruction is accessible and inclusive. 

The UDL concept is the philosophical foundation 
for inclusive teaching, and the literature demonstrates 
that many inclusive strategies are effective. Orr and 
Hammig’s (2009) survey of pedagogical techniques 
found that in 21 of 38 studies, inclusive techniques 
and learner supports were in use. These studies pro¬ 
vide evidence for the ability of inclusive instruction to 
minimize the need for students with disabilities to seek 
formal accommodations. Nevertheless, there is room 
for further inquiry. Despite its increasing deployment 
in K-12 education, UDL is not as widely implemented 
in postsecondary education. One broad conclusion 
gleamed from a review of the scholarly literature is 
a shortage of research on UDL and accommodations 
as they apply to the university setting (Moon, Todd, 
Morton, & Ivey, 2011). 

SciTrain University 

SciTrain U is designed to enhance the capacity of 
university STEM faculty and staff to improve learn¬ 
ing for all students, including those with disabilities, 
through the application of UDL practices. In doing 
so, this project relies on two major components. First, 
in-person workshops are delivered with the intent of 
educating faculty about disability awareness and work¬ 
ing with students with disabilities. In accord with the 
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project’s UDL emphasis, workshops have focused less 
on disability accommodations and more on using ac¬ 
cessible pedagogy to improve learning outcomes for all 
students. The workshop developer has given particular 
attention to three STEM learning environments: large 
lecture classrooms, laboratories, and online learning 
environments. For each of these, workshops have 
focused on multiple approaches to make learning 
more effective for all students. For example, group 
note-taking activities and personal response systems 
(PRS, or “clickers”) are discussed as means to improve 
instructional outcomes in lecture-based courses, while 
the development of inquiry-based labs are discussed 
for lab-based courses. 

Workshops are developed and delivered by a lead 
instructional technologist at Georgia Tech’s Center for 
the Enhancement of Teaching and Teaming (CETF). 
The workshop developer has almost 15 years of expe¬ 
rience in distance education and has worked closely 
with faculty regarding the use of technology in con¬ 
ventional and virtual classroom settings. The workshop 
developer has collaborated with biology and marine 
sciences faculty at the University of Georgia and 
chemistry and applied physiology faculty at Georgia 
Tech to ensure the incorporation of appropriate STEM 
content knowledge within SciTrain U workshops. 
Workshops on different topics are offered at each of 
the two participating campuses three to four times per 
semester; they are occasionally offered more than once 
whenever interest is relatively high. As the project has 
progressed, the workshop developer has revised the 
face-to-face workshops in response to survey findings 
as well as to update content. For more information 
about workshops, please visit http://www.catea.gatech. 
edu/scitrainU/login.php. 

Building upon the workshops, the project website 
hosts online courses for deployment of project resourc¬ 
es at other institutions. Considerations of accessibility, 
usability, simplicity of design, and visual appeal drive 
the website’s design, as does clear “branding” for the 
project, simplified page layouts and functionality, 
and deliberate use of high contrast and white space. 
The site provides STEM instructors with three types 
of content: background information on common dis¬ 
abilities, an overview of disability accommodations, 
and information about inclusive pedagogy. The site 
provides modules on transitioning from secondary 
schools, introduction to UDL, learning disabilities, 
attention-deficit/hyperactivity disorder (ADEID), au¬ 


tism, mobility and dexterity disabilities, deafness and 
hearing impairments, low vision and blindness, and 
disability laws. In illuminating the types of disabilities 
that instructors might encounter, the modules discuss 
a variety of methods to accommodate learning needs. 
The web materials also address assistive technologies 
and how best to integrate their use within classroom 
and laboratory learning. 

Evaluation and Assessment 

To document SciTrain U’s effectiveness, consider¬ 
able project resources are allocated for an evaluation 
and assessment team. Two lead evaluators are based 
at Georgia Tech’s CETL and Center for Advanced 
Communications Policy (CACP), and their team has 
generally included one undergraduate assistant and up 
to three graduate assistants at any given time. While 
many of the resources associated with the project typi¬ 
cally have been based at CETL, two of the graduate 
assistants are closely associated with the disability 
services centers at Georgia Tech and UGA. 

Evaluation Methodology and Approach 

Synthesizing data is often difficult for evaluation 
teams unless a specific approach is utilized. This evalu¬ 
ation team’s methodology involves combining two 
mixed-methods approaches for analyzing data: Mc- 
Conney, Rudd, and Ayres’ Results Synthesis Method 
and Campbell’s Pattern Matching method. The Pattern 
Matching method recommends that evaluators work 
from a model such as a program logic model and 
identify whether each aspect of the model enables or 
prevents the program from reaching its intended im¬ 
pact (McConney, Rudd, & Ayres, 2002). The Results 
Synthesis Method guides evaluators in working with 
stakeholders to identify the value of each evidence 
set, such as classroom observations and focus groups 
(Campbell, 1966). This allows the evaluators to more 
accurately depict the strength or weakness of the ties 
among each block on the logic model. 

Aside from the analysis of performance data on 
students enrolled in project-affiliated courses, SciTrain 
U’s evaluation has rested mainly upon qualitative 
methods, including classroom observations, faculty 
workshop surveys, student surveys, website surveys, 
online journals, and focus groups. Where observational 
methods are utilized, multiple evaluators have taken 
part to insure interrater reliability. Likewise, all open- 
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Figure 1. SciTrain U Logic Model. 


ended responses to surveys and online journal entries 
have used at least two coders to maximize reliability. 

Indicator Selection 

Scholars concerned with the evaluation of postsec¬ 
ondary STEM education have observed that the basic 
availability or, more frequently, unavailability of data 
should not dictate the approach for undertaking evalu¬ 
ations (Coates, 2007). These same principles hold true 
for programmatic interventions to support students with 
disabilities. If evaluation is to determine educational 
efficacy, then it must be considered from the beginning 
through the development of meaningful indicators and 
the provision of data collection and analysis for such 
ends. With this guiding principle in mind, evaluators 
sought to determine the appropriate indicators at the 
outset of the project and then formulated data col¬ 
lection efforts in response. Evaluation questions and 
corresponding indicators emerge from the logic model 


(see Figure 1), which governs the evaluation process 
and details the prospective inputs, project activities, 
objectives, outcomes and goals, and impact. 

Evaluation of SciTrain U provides data for forma¬ 
tive and su mm ative feedback along three key tracks. 
In the first track, the Referenced Program Performance 
Indicators specifically address the U.S. Department 
of Education’s Government Performance and Results 
Act (GPRA) Goal #3, Ensure the accessibility, af¬ 
fordability, and accountability of higher education, 
and better prepare students and adults for employ¬ 
ment and future learning. The referenced program 
performance indicators provide critical information 
on student course completion patterns and on faculty 
adoption of SciTrain U resources. The second track, 
Knowledge Synthesis, combines SciTrain U’s research 
findings with institutional knowledge developed from 
prior projects. These include SciTrain: Science and 
Math for All, which is oriented at improving science and 
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Table 1 

SciTrain U Evaluation Questions 


Track 1: Referenced Program Performance Indicators 


a. What is the rate at which students with documented disabilities complete courses taught by SciTrain 
U-trained faculty? What is the rate at which students without documented disabilities complete courses 
taught by SciTrain U-trained faculty? What, if any, is the difference in these two rates? 

b. What percentage of SciTrain U-trained faculty incorporates elements of training into their 
classrooms? 

Track 2: Knowledge Synthesis 


Based on current literature and recent SciTrain and GRADE findings, what learning environments will prove 
most successful for program stakeholders? 

Track 3: Program Implementation Measures 


a. Which stakeholders are provided with what resources? 

b. What do participants learn as a result of program participation? 

c. What actions are the various stakeholders taking toward improving content/pedagogical knowledge, 
organizational capacity, and available resources? 

d. Who adopts which aspects of inclusive instruction? 

e. What organizational barriers or accelerators hinder or promote inclusive instructional practice 
adoption? 


mathematics instruction for students with disabilities in 
secondary education, and Georgia Tech Research on Ac¬ 
cessible Distance Education (GRADE), which focuses 
on accessibility of distance education for students with 
disabilities. The third track, Program Implementation 
Measures, provides necessary formative assessment 
feedback for program benchmarking and improvement 
as the program is being implemented. This track cul¬ 
minates with a loop back to Track 1. The process for 
each track is addressed by the project evaluation team 
through the use of a set of targeted evaluation questions 
designed to elicit meaningful data collection and analysis 
(see Table 1). 

The GPRA-derived indicators comprising Track 
1 were ascertained through data collection efforts 
involving longitudinal participation by affiliated fac¬ 
ulty at Georgia Tech and UGA. In order to determine 
completion rates of students with disabilities taught by 
SciTrain U-affiliated instructors, evaluators analyzed 


demographic and performance data collected by the 
disability services centers and registrar’s offices at 
both institutions. In order to establish the percentage 
of SciTrain U-trained faculty incoiporating elements of 
training into their classrooms, as well as the nature of 
those practices, longitudinal participants participated 
in classroom observations and kept online journals 
detailing their experiences in the project. 

Instrument Creation and Use 

A “development” mixed-methods approach was 
used to generate evaluation instruments, includ¬ 
ing workshop surveys, focus group protocols, and 
classroom observation instruments (Greene, 2007). 
Information from a baseline literature review informed 
the evaluation team in its development of these instru¬ 
ments. The knowledge synthesis effort also resulted 
in a list of characteristics the program expected to see 
transferred from instruction into practice. This “trans- 
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fers list” assisted in the creation of these instruments. 
Also, Greene’s “complementarity” mixed-methods 
approach was applied, which means that the class¬ 
room observations served to complement other data 
collection efforts by providing a deeper description 
of how faculty apply what they have learned into their 
classrooms and also to confirm or disconfirm the self- 
reported information from surveys and focus groups 
(Greene, 2007). 

Longitudinal Participants 

In order to evaluate the effectiveness of the project, 
longitudinal participants were recruited from both par¬ 
ticipating institutions. A total of 15 faculty members, 
nine from Georgia Tech and six from UGA, were re¬ 
cruited for this project. Collectively, they represented 
departments of biology, chemistry, mathematics, as¬ 
tronomy, marine sciences, and applied physiology (i.e., 
health). In order to ensure their continued involvement 
for the duration of the project, the longitudinal partici¬ 
pants were compensated with modest stipends, as well 
as additional honoraria for taking part in supplemental 
project activities. As discussed below, their retention 
for the duration of the project posed a challenge. Nev¬ 
ertheless, most of the participants continued for at least 
four terms, or two years, of the project period. 

Relying upon a mixed-methods approach, three 
types of data specific to longitudinal participants 
were collected in order to evaluate SciTrain U’s ef¬ 
fectiveness and document change-over-time impacts. 
First, performance and demographic data for enrolled 
students with disabilities were collected in order to sat¬ 
isfy the GPRA-derived indicators. Second, classroom 
observations were undertaken twice a term for each 
of the participants in order to assess the accessibility 
of classroom and laboratory instruction. Third, par¬ 
ticipants were required to submit entries for an online 
journal that recorded their experiences in the project. 
In addition to these data collection efforts, participants 
also attended workshops. 

Performance and Demographic Data 

In order to determine the effectiveness of SciTrain 
U as a programmatic intervention to improve the learn¬ 
ing outcomes of students with disabilities, evaluators 
collected both performance and demographic data on 
students with documented disabilities enrolled in SciTrain 
U-affiliated courses. These data are collected in order to 


determine the rate at which students with documented 
disabilities complete courses taught by faculty trained 
in project activities, and the rate at which other students 
without disabilities complete those courses. 

In order to assess these outcomes, evaluators devel¬ 
oped collaborative partnerships with the disability ser¬ 
vices centers and registrar’s offices at both participating 
institutions. Graduate research assistants tasked to the 
disability services centers at Georgia Tech and UGA 
facilitate identification of students with documented 
disabilities enrolled in project-related courses. Be¬ 
cause of a lack of systematized data reporting at these 
institutions, a project-specific spreadsheet instrument 
was developed that allows identification numbers for 
all students on file with disability services to be com¬ 
pared against enrollment rosters for each of the courses. 
Once students are identified, queries are submitted to 
the registrars to provide pertinent demographic and 
performance data immediately following the end of 
the term. Demographic data gathered include students’ 
gender, race, disability, class standing, and major, and 
performance data includes course grades (including 
incompletions and withdrawals), semester grade point 
averages (GPA), and overall GPA. 

In spring 2009, the evaluation team collected its 
first set of performance data from the courses of four 
longitudinal participants at Georgia Tech. This dataset 
established a baseline for evaluating subsequent prog¬ 
ress, and it revealed that in courses taught by SciTrain 
U-trained faculty, 94.45% of students with disabilities 
(17 of 18) successfully completed the courses in ques¬ 
tion. The one student who withdrew did so due to 
circumstances not related to the student’s disability or 
academic performance in the course. More specifically, 
88.24% of students who completed a course under evalu¬ 
ation (15 of 17) received a passing grade in the course 
(Grade distribution: A=9; B=4; C=2; D=0; F=2). More 
widely, 16 of the 18 students evaluated were in good 
academic standing, with one on academic warning and 
another on probation at the time. 

In fall 2009, there were a total of 21 students with 
documented disabilities across six courses at Georgia 
Tech taught by SciTrain U-trained faculty. In the Math 
1711 course, there were four students with disabilities. 
Their average grade for the course was a B- (2.75) (uti¬ 
lizing a four-point grading scale, A=4, B=3, C=2, D=l, 
F=0). Math 1113 had one student with a disability, who 
earned an A in the course. The first section of Chemistry 
1510 had three students with disabilities, with an aver- 
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age of C+ (2.33), while the second section of Chemistry 
1510 had one student who earned a C in the course. The 
first section of the freshman-level health/wellness semi¬ 
nar, Health Performance Science (HPS) 1040, included 
three students with disabilities, with an average of a B 
(3.00) in the course. The second section of HPS 1040 
included 9 students with disabilities, with an average of 
a B (3.00) in the course. In addition, overall GPAs and 
semester GPAs were tracked. 

The comparison between fall 2009 and the spring 
2009 baseline suggests a complementary set of conclu¬ 
sions. First, while a course-by-course comparison may 
not reflect an improvement in student performance, 
overall, the project does appear to be making a broader 
impact in terms of the numbers of students with docu¬ 
mented disabilities reached, their course completion 
rates, and passing grades earned. As noted, the number 
of students impacted by the project at Georgia Tech to 
date, in terms of longitudinal participants and students 
enrolled, makes statistical significance difficult. This 
suggests a second point, which is that qualitative data 
may be just as relevant as indicators of student success 
as performance data. 

Classroom Observations 

Drawing upon the scholarly literature on UDL 
approaches to postsecondary classroom instruction 
(Fahsl, 2007; Fuller, Bradley, & Healy, 2004; Higbee, 
2003; Orr & Hammig, 2009), project evaluators devel¬ 
oped a classroom observation instrument (see Figures 
2 and 3) to conduct beginning-of-term and end-of-term 
observations for each longitudinal faculty participant. 
This 48-item instrument considers six elements of 
inclusive and accessible pedagogy: classroom envi¬ 
ronment, visual aids, oral communication, “clickers” 
(i.e., electronic personal response systems), classroom 
notetakers, and electronic learning support (i.e., course 
management software). Observations are made by two 
raters, which always include at least one of the two lead 
evaluators. Graduate research assistants tasked to the 
project also participated in observations, and training 
was provided in person and reiterated through the de¬ 
velopment of an evaluation team manual. During an 
observation, observers mark the item as “Yes,” “No,” 
or “N/A,” depending on whether the behavior was 
observed. An affirmative answer is generally meant to 
indicate that the instructor adheres to the principles of 
UDL, where a negative generally suggests that such 
adherence was not observed. While the polar nature of 


the observation form permits scoring, both individu¬ 
ally and as a group, notetaking is also done to provide 
clarification and feedback for participating instructors, 
as well as to allow for more detailed explanations or 
descriptions of the observations. 

Faculty participants are observed twice per term, 
and the same two scorers are involved in both begin- 
ning-of-term and end-of-term observations. (In order 
to expedite scheduling of observations and avoid po¬ 
tential conflicts with tests or special activities, faculty 
participants were provided with advance notice of the 
days they were observed.) Use of the same observers, 
as well as discussion of findings at the end of each ob¬ 
servation to resolve any inconsistencies, ensures some 
degree of reliability. The team also has developed an 
observation guide that accompanies the instrument, 
and it contains an item-by-item explanation to assist 
raters in making their observations. Both the instru¬ 
ment and guide have been subject to periodic review, 
and slight refinements have been made in response to 
prior experiences. 

As of fall 2010, evaluators had completed 80 
observations of 15 longitudinal participants. Four 
of the participants had been involved continuously 
throughout the three terms of the study, and live of 
the participants had been involved for two terms. At 
the time of the analysis, the remaining six participants 
only had a baseline measure. As previously noted, 
the instrument consists of 48 items (three items were 
simple counts and 45 were categorical) that probe six 
aspects of instruction. A corresponding “accessibil¬ 
ity score” is derived from the 45 coded items. The 
scores are a sum/composite of these items, allowing 
for a maximum of 45 and a minimum of -45. “Yes” 
responses are coded as a 1, and “no” responses are 
coded at a -1, while “N/A” responses are coded as 0 {Y 
= 1, N = -1, N/A= 0}. A corresponding scoring rubric 
was devised, in which a score of < 0 denotes “poor,” 
1-15 is “fair,” 16-30 is “good,” and 31-45 is “excel¬ 
lent.” The underlying rationale for the scoring rubric 
was that any score below 0 was undesirable from an 
accessibility standpoint. 

The following graph (see Figure 4) presents long¬ 
standing participants’ accessibility scores averaged 
across all their observations. A corresponding graph 
(see Figure 5) shows the aggregate change-over-time 
results from the observations. The regression line sug¬ 
gests a trend of increasing accessibility scores over the 
project’s course. When examining accessibility across 
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Classroom Observation Form 
Date: 

Time Start: End: 

Teacher: Course: Student # 



Classroom Environment 

Closes door and/or blinds 
Welcomes or greets students 
Reminders given about electronics during class 
Reminders given about acceptable classroom etiquette 
Action taken to motivate students in class or in general 
Language used does not stereotype students 
Flexibility to address individual needs demonstrated 
Students provided with multiple ways to learn 
Content is made personally relevant to student lives 

Visual Aids 

Class outline presented/provided 

Handouts provided 

Handouts highly readable 

Materials easily visible from back of classroom 

Materials uncluttered 

Materials well organized 

Variety of types of visual aids used 

Number of Student questions on visual aids:_ 

Clarity:_ Comprehension:_ 

Oral Communication 

Uses student names 

Faces class to speak when not writing on board 
Clearly audible from back of room 
Makes eye contact while speaking to students 
Clearly explains visual aids 
Gives clear instructions for student activities 
Instructions for student activities repeated 
Student interaction actively facilitated 
Summarizes major points 

Number of Student questions in general:_ 

Clarity:_ Comprehension:_ 


Y 

N 

N/A 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Y 

N 

N/A 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Y 

N 

N/A 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 


Notes: 


Figure 2. SciTrain U Classroom Observation Instrument, Page 1. 
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SGfhKA+N e 


Teacher: 


Course: 


Clickers 

Y 

N 

N/A 

“clickers” are used in the classroom 

□ 

□ 

□ 

If used, students are able to easily connect 

□ 

□ 

□ 
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Classroom Notetakers 

Y 
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Class note takers used 

□ 

□ 

□ 

Offers printed materials that facilitate note taking 

□ 

□ 

□ 

Gives reminders of important points to include in notes 

□ 

□ 
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Gives appropriate pauses for students to take notes 

□ 

□ 

□ 

Offers feedback or instruction on good note taking 
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□ 

□ 

Electronic Learning Support 

Y 

N 

N/A 

Majority of students come to class with proper materia 

ls D 

□ 

□ 

Online communication with instructor encouraged 

□ 

□ 

□ 

Online communication with other students encouraged 

□ 

□ 

□ 

— to be observed outside of the classroom — 




Materials available at least 24 hours before class 

□ 

□ 

□ 

Materials provided in accessible format(s) 

□ 

□ 

□ 

Online communication with instructor facilitated 

□ 

□ 

□ 

Online communication with other students facilitated 

□ 

□ 

□ 

Options for students to post materials for class 

□ 

□ 

□ 

Options for students to post their own materials 

□ 

□ 

□ 

Lectures available by audio file 

□ 

□ 

□ 

Lectures available by video file 

□ 

□ 

□ 

Online audio/video materials clear/usable 

□ 

□ 

□ 

Students know when recordings will be available online 

□ 

□ 


Classroom Diagram 


Figure 3. SciTrain U Classroom Observation Instrument, Page 2. 
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the six sections of the instrument form, the increasing 
accessibility of class notetaking and electronic learn¬ 
ing support, by term, corresponded to when workshops 
were held on these areas. When examining standardized 


change in average accessibility by section, the analysis 
revealed improvements in class notetakers, oral com¬ 
munication, visual aids, and electronic learning support, 
in that order (see Figure 6). 


Average Accessibility Score of Multi-Term 

Participants 

45 

40 

35 



Figure 4. SciTrain U Longitudinal Participant Accessibility Scores, Averaged Across All Classroom 
Observations. 


All Accessibility Scores across Time 



Figure 5. Change over Time of SciTrain U Longitudinal Participant Accessibility Scores. 
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Standardized Change in Averaged Accessibility 
across Participation for Instrument Sections 



Class Oral Visual Aids 

Notetakers Communication 



Electronic Classroom 

Learning Environment 

Support 



Figure 6. Standardized Change in Accessibility Scores for Each of the Six Instrument Sections. 


Online Journal Reflections 

Longitudinal respondents were also required to 
maintain weekly online journals. Each journal entry 
involved a response to an open-ended question regarding 
the relationship between the learning environment in the 
participant’s classroom and students with disabilities. 
Group workshops conducted in parallel to the journal 
responses during the spring 2009 term addressed the 
use of online forums in the classroom, while workshops 
during the fall 2010 term focused on the use of online 
forums and collaborative learning activities in the 
classroom. For this reason, the majority of participant 
responses for spring 2009 and fall 2010 were concerned 
with the educational effect of online forums and group 
learning methods, respectively. Journal entries articulate 
teaching methods, instructor concerns, student feedback, 
implementation challenges, and overall learning. The 
journal entries also discuss specific changes and de¬ 
velopments related to all students, particularly students 
with disabilities. 

Analysis of faculty participant responses are 
conducted based on the following questions: 1) What 
aspects of inclusive instruction do faculty participants 
adopt? 2) What barriers or accelerators hinder or pro¬ 
mote inclusive instruction practices? 3) What actions 
have faculty participants been taking toward improv¬ 
ing content/pedagogical knowledge, organizational 
capacity, and available resources? 4) What do faculty 
participants leam as a result of the adoption of certain in¬ 
clusive teaching methods? 5) What particular resources 


and accommodations are provided to SWDs and how 
are they effective? 

In order to answer those questions, evaluators em¬ 
ploy a qualitative method to analyze pertinent responses. 
The main analytical procedure is to do preliminary 
coding (e.g., open coding), draw out some themes and 
concepts, and re-code to develop better defined catego¬ 
ries and analyze data. This coding is spot-checked by 
a second researcher at least twice during the analysis 
process. Journal reflections are analyzed with three 
focuses: overall themes, change over time, and par¬ 
ticipant archetypes. By way of example, we present an 
overview of findings from spring 2010, during which 
eight participants submitted a total of 44 journal entries. 
They are referred to below as Participants A, B, C, D, 
E, F, G, and H (see Table 2). 

Entries were first examined to determine overall 
themes. Discussions of teaching methods were most 
prominent, particularly group learning methods that 
were deployed in tandem with SciTrain U workshops 
on the topic. Five out of the eight faculty participants— 
Participants A, B, F, H, and G—used various group¬ 
ing and interactive learning mechanisms as tools for 
promoting students’ learning through collaboration. 
The variety of formats included group note-taking, 
group assignment, group class presentation, in-class 
group work, group-based field experience, and group 
tests. In addition to group-based learning, participants 
tried developing better class materials and using other 
teaching methods in order to promote students’ leam- 
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Table 2 

Summary of Journal Entries by SciTrain U Longitudinal Participants, Spring 2010 


Number 

Participant of Journal 
Entries 


Course 

Subject 


Main Inclusive Instruction Practices 
Used 


School 


A 


B 

C 

D 

E 

F 

G 


H 


1 


6 

4 

6 

3 

2 

5 


17 


Group note-taking, Group testing, 
Lecture podcast, Clickers, Online 
Biology study materials, Posted PowerPoint 

(PPT)-slides, Extra exam time, Peer 
review evaluation 

^ Group note-taking and assignment, 

Peer review evaluation 


Health 


Annotations, Online forum, Online 
chat room 


Online study materials, Online forum, 
Biology Annotations, Adjusting physical 
environment 


. Extra exam time, Posting extended 

Chemisty 

notes, Lecture podcast 


. Group works m class, Color-coding 

Astronomy 

the contents on ppt-slides. 


TT ,, Group note-taking, Peer review 

Health , . 

evaluation 


A think-share-pair, Group works and 
assignments, Peer review evaluation, 
Biology Field trip, Online forum, Inviting guest 
speakers for lab activity, A weekend 
work day 


UGA 


GT 

GT 


UGA 


GT 

UGA 

GT 


UGA 
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ing. Participants A, E, and D provided online study 
materials, including animations, interactive tutorials, 
video clips, newspaper articles, PowerPoint lecture 
slides, and handouts. Other strategies to improve stu¬ 
dent learning included open-book, pre-lecture quizzes, 
lecture podcasts, making lecture notes available online, 
enhancement of classroom discussions through online 
forums and chat rooms, and the use of other instruc¬ 
tional technologies. 

Another theme involved student receptivity to 
these inclusive instructional methods. Most faculty par¬ 
ticipants reported that students gave positive feedback 
in response to their group-based learning. Participant A 
cited that about 90% students recommended continu¬ 
ing group tests in the end-of-semester survey because 
“group test helped them learn the material more ef¬ 
fectively, made the class less impersonal, and helped 
them enjoy the class more.” By contrast, some students 
complained about the group test/work, especially when 
they felt that their group was dysfunctional. In the 
same manner, Participant G found that the majority 
of students felt that group note-taking had an impact 
on improving their exam score. In the survey, students 
also remarked that “the group assignment made them 
attend class, read the information weekly, so that they 
stayed on track.” Participant G cited one student’s 
comment about the positive effect of group-based 
learning, representing that students learned how to 
cooperate with others by doing the group assignment. 
Near the end of term, Participant H contended that 
students had improvements in their public speaking 
skills as students reported being far more comfortable 
in front of class. 

Students also responded to overall class materials 
and learning. Faculty participants A and D reported 
that students’ comments about online course materi¬ 
als were positive as they found it useful. Participant A 
implemented an open-book pre-lecture quiz in order 
for students to learn those materials, but some students 
complained about this pretest. While students wanted 
to take the exam after all materials were covered in 
class, the instructor felt that students had been better 
prepared for class after using pre-tests. Participant C 
reported that many students had interest in using the 
online forum in that 70% of students posted at least 
once, and more than 50% posted five or more times 
during the semester. In the survey, many of the students 
answered that the use of the online forum for class dis¬ 
cussion was effective in promoting their understanding 


of the course materials and suggested to continue the 
use of the forum. 

Each faculty participant made comments specific 
to their students with disabilities. Participants A, G, 
and E provided students with documented disabilities 
extra exam time in a quiet room for individual tests 
in addition to accommodations already provided. 
Participant E generally allowed all students to have 
extended time to finish their exam, because “extra 
time let students have some time to think through the 
questions rather than to rush through and regurgitate 
facts that they have learned.” Participants C, D, and F 
expressed concern about visual aids and materials for 
their students with visual impairments. More specifi¬ 
cally, Participant C was worried about the graphical 
nature of online course materials, since they are very 
visual, while Participant D raised the serious issue of 
a poorly arranged classroom: 

There is insufficient control of the amount of 
natural light entering the room...the fluorescent 
lighting cannot be controlled at a sufficiently 
fine-grained level.. .when it is turned off, it is too 
dark to clearly read the chalkboard in much of the 
room.. .1 can’t imagine what it would be like for 
those who are visually impaired. 

Participant F used color-coding for information 
that was left off of posted class slides so that students 
could easily identify what is missing, assuming that 
this would be a disadvantage for colorblind students. 
Interestingly, there were no students with identified 
visual impairments enrolled in the class. 

Four of the faculty participants (B, D, G, and H) 
have been enrolled in the SciTrain U project since its 
inception, allowing evaluators to document change¬ 
over-time impacts through analysis of the online 
journals. For example, Participant B noted positive 
experiences with the group note-taking assignments 
in fall 2009, especially where student engagement was 
concerned. This experience, however, shifted after the 
participant faced a cheating issue in a subsequent term, 
when one group downloaded other groups’ notes and 
re-posted it as their own work. Although Participant 
B still believed the effectiveness of the group-based 
learning, preventing student cheating became a prereq¬ 
uisite and, hence, an implementation challenge for con¬ 
tinuing this method. In addition to warnings about the 
consequences of academic dishonesty, the participant 
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added a peer review component to the grade. After using 
the peer evaluation system in spring 2010, Participant B 
found that peer evaluation was not as effective as hoped. 
Many groups gave perfect scores to the members of the 
team regardless of their contributions. 

Finally, journal entries were analyzed to discern 
participant-specific development in order to determine 
whether any archetypes emerged as a result of their 
participation in the project. Three broad categories of 
faculty participants were identified: Enthusiast, Skep¬ 
tic, and Incremental Adopter. During the spring 2010 
period, both Participants B and H were recognized as 
Enthusiasts based on their interest in further develop¬ 
ing the group-based learning as effective pedagogy for 
promoting student learning. Participant B demonstrated 
particular enthusiasm for implementing group note¬ 
taking and group peer evaluation and ended the term 
with plans to improve the pattern of group study and 
add group projects to grade components in the future. 
Participant H found group-based learning to be very 
effective throughout the one-year period of study. By 
contrast, Participant D was a consistent Skeptic about 
the use of online forums to enhance classroom learn¬ 
ing for the two semesters. This instructor encouraged 
students to discuss materials online, but very limited 
student participation led the instructor to answer most 
of the questions. In addition, the participant found 
that the online forum lacked the tools necessary to 
describe and write mathematical equations, a techni¬ 
cal issue that has yet to be remedied. Falling between 
the archetypes of enthusiast and skeptic, Participant G 
represented the Incremental Adopter. This instructor 
found that prescribed group-based learning appeared 
to work well, as the majority of students commented 
that group notes were useful. However, the participant 
showed mixed feelings about the effect of the meth¬ 
ods. While the quality of group note-taking as well as 
interaction in the class was improved, this instructor 
was not sure if this had a positive impact on students’ 
grades. Participant G noted several implementation 
challenges stemming from the large, lecture-based 
course in which the approach was used. Despite these 
concerns, this instructor ended the term with plans to 
continue refining the methods in order to make them 
suitable for the class format. 


Limitations and Challenges 

While this article calls attention to the successes 
of SciTrain U through a discussion of its evaluation 
methodology and findings, the project was beset by a 
number of challenges. In addition to evaluating project 
outcomes, evaluation personnel were also tasked with 
internal assessment in order to determine challenges 
in need of resolution. In this article, we highlight two 
challenges in particular: relatively low enrollment by 
students with documented disabilities and difficulty 
securing longitudinal participants and faculty cham¬ 
pions. While these challenges persisted throughout 
the project, the leadership team attempted to resolve 
these and other issues identified through the internal 
assessment process. 

Internal Project Assessment 

Internal assessment is accomplished through 
the application of the SII (Strengths, Areas for Im¬ 
provement, and Insights) model as part of internal 
quarterly reporting (Wasserman & Beyerlein, 2007). 
These reports are used to summarize evaluation ac¬ 
tivity, identify effective programmatic results from 
which synergy can be built in related areas, pinpoint 
areas of immediate need along with specific advice 
to address those needs, and provide data that may be 
generalized to similar program efforts elsewhere. The 
SII reporting model is used to provide periodic ongoing 
assessment of program activities and the evaluation 
process itself. 

Patton (1997) challenges evaluators to understand 
that evaluation use must be facilitated and emphasizes 
that it rarely, if ever, happens by chance. To ensure 
that the evaluation plan and findings provided useful, 
actionable information, the evaluation team presented 
timely information to allow for program modifications. 
As evaluation findings through the SII process have 
surfaced, the principal investigator and other personnel 
have worked to address identified challenges. Some of 
these improvements have included modifications to 
curriculum, instruction, and delivery methods; changes 
in technical assistance approaches and other dissemina¬ 
tion methods; targeting specific support communities 
for more extensive training and assistance; and iden¬ 
tifying potential new resources and partnerships not 
currently apparent. As these challenges are resolved, 
some have come to constitute project strengths. 
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Student-Side Engagement 

SciTrain U is designed primarily as an instructor- 
oriented project in the sense that faculty training rep¬ 
resents its main focus. Nevertheless, engagement of 
students, especially those with documented disabilities, 
remains fundamental. More practically, the enrollment of 
students with disabilities in SciTrain U courses is vital for 
evaluation of the project’s efficacy. Enrollment was lower 
than expected during the first two years, complicating the 
generalizability of performance evaluations. Relatively 
low numbers of students in SciTrain U-affiliated courses 
undermined the statistical significance of data gathered 
through the evaluation process. In addition, there was a 
more fundamental need to determine the broad impact of 
the project on students. 

In order to address recruitment of students, par¬ 
ticularly at Georgia Tech, a GRA was tasked to the 
disability resource center. In addition to gathering 
demographic and performance data on behalf of the 
evaluators, the GRA served as a student liaison. Dur¬ 
ing the summer, the head of disability services and the 
GRA held events at all six of the freshman and transfer 
orientation sessions. They met with incoming students 
with disabilities and parents regarding SciTrain U and 
the possible benefits it could offer, as well as pre-reg¬ 
istering any students expressing an interest. As a result 
of these efforts, enrollment in project affiliated courses 
has grown substantially. Though the data have yet to 
be analyzed, a total of 44 students were enrolled for 
the fall 2010 term, substantially more than the baseline 
enrollment of 18 students with disabilities. 

An online survey instrument for evaluating stu¬ 
dent perceptions of the accessibility of SciTrain U 
courses was also developed and distributed. Collecting 
self-report data from students enrolled in the courses 
of participating faculty, the survey gauges student 
perceptions about inclusion within the university and 
classroom environment as well as the accessibility of 
instructional methods and materials. While written 
ostensibly to obtain feedback from students with dis¬ 
abilities, the survey was designed to be administered 
meaningfully to all students in a course. Another key 
rationale of the survey is to collect data that can be 
roughly correlated to the findings of SciTrain U’s class¬ 
room evaluation instrument. While this survey does not 
ask the same specific questions as that instrument, it 
broadly probes the same areas: the physical classroom 
environment, professor awareness of student needs, 
written materials (i.e. textbooks, course packets, hand¬ 


outs), oral communications (i.e., lectures, discussions), 
and evaluation of student learning (tests, exams). 

While such student feedback has the disadvantage 
of not probing specific items of interest in the SciTrain 
U classroom evaluation instrument, the instrument 
does confer a number of potential benefits. First, it 
provides some measure of the impact of SciTrain U 
on students themselves. While evaluators may assess 
through classroom observation how well instructors 
are putting workshop and online course module les¬ 
sons into practice, this instrument provides a means 
for comprehending, if only in a rudimentary fashion, 
what such practices mean for the student. As this sur¬ 
vey can be given at both the beginning and end of a 
course, it is also possible to gauge change over time. 
Second, the instrument is open-ended so that students 
can elaborate on issues that are important to them, 
allowing direct evaluation of programmatic impact 
on STEM education for students with disabilities and 
indirect evaluation of the program’s impact on all stu¬ 
dents. While student-self reporting was not listed as an 
original evaluation tool, this instrument and its findings 
will help augment the evaluation of the effectiveness 
of SciTrain U through its correlation with classroom 
observation findings and by probing other dimensions 
of the program. 

SciTrain U Scholars Program 

One persistent challenge identified through the SII 
process has been a need for more faculty involvement 
in the SciTrain U project, especially at Georgia Tech. 
To address this need, project leadership developed 
the SciTrain U Scholars program to improve outreach 
through the use of faculty champions. A total of eight 
faculty members at Georgia Tech participated in the 
SciTrain U Scholars program during fall 2010. Among 
their primary activities, the group was tasked with mak¬ 
ing faculty contacts on behalf of the project (including 
individuals to pilot the online materials), giving pre¬ 
sentations on their activities, and providing feedback 
on their involvement with the project. 

A total of 32 contacts were made during this period, 
including six tenure-track faculty, eight non-tenure 
track faculty (i.e. academic professionals), and nine 
teaching assistants. Of this number, 27 were confirmed 
as online course participants. The main departments 
represented in these activities included mathematics, 
biology, mechanical engineering, aerospace engineer¬ 
ing, computer science, applied physiology (i.e. health/ 
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wellness), and learning services. Also, a total of 14 
presentations were made, including five department 
meetings and three external conferences. 

Scholars noted that the online tutorials provided 
significant feedback on their teaching and led to the 
adoption of more inclusive teaching methods in many 
cases. In terms of engagement with administration, they 
reported some success, including school chairs, curricu¬ 
lum committees, and several deans. When asked about 
challenges, a lack of time was the overwhelming response. 
In particular, there was a constant call to shorten the online 
course modules. Budget cuts and the continued lack of 
involvement by tenure-track faculty were also identified 
as challenges. Despite these issues, however, the project 
leadership has found this program to be relatively suc¬ 
cessful in boosting outreach efforts. 

Discussion and Relevance for Practitioners 

As a large-scale project designed to support and 
enhance postsecondary STEM education for students 
with disabilities, SciTrain U is representative of similar 
projects sponsored by National Science Foundation’s 
(NSF) Research in Disabilities Education (RDE) and 
the U.S. Department of Education’s Office of Postsec¬ 
ondary Education (OPE), and other federal agencies. 
Such projects are mandated to demonstrate their ef¬ 
ficacy and potential for improving outcomes for these 
students, yet there is relatively little published on the 
evaluation of such projects. The evaluation approach 
discussed in this article may be of use for investiga¬ 
tors seeking novel means to discern the effectiveness 
of these projects. 

In order to maximize evaluation efforts, our project 
took a mixed-methods approach that may be relevant 
for similar projects. Several of our instruments are 
now being deployed at the beginning and end of each 
term. In addition, reliability is enhanced, where pos¬ 
sible, through the use of multiple raters. The use of 
these instruments has allowed for data triangulation, 
whereby the various instruments provide different 
perspectives of the same project element under con¬ 
sideration. Feedback forms and focus groups provide 
unique insights into the workshops, while classroom 
observations, student surveys, and online journal re¬ 
flections allow for a multi-perspective examination of 
longitudinal participation. In short, our use of multiple 
instruments that permit for triangulation has facilitated 
richer data analysis. 


Conclusion 

As a case study for the evaluation of programmatic 
interventions to enhance postsecondary STEM educa¬ 
tion for students with disabilities, the authors believe 
that SciTrain U contributes to scholarship and practice. 
The multi-faceted approach taken by the project evalu¬ 
ators, characterized by a mixed-methods approach that 
documents project effectiveness through longitudinal 
participants, may be of use for similar projects. Given 
a dearth of scholarship on the evaluation of programs 
to enhance STEM education for students with disabili¬ 
ties, this article seeks to provide some insights into the 
potential for mixed-methods approaches. 
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